This invention relates generally to a system for importing data into a data analysis system and, more particularly, to a system for importing data into a data analysis system for graphically displaying data relationships for the imported data.
To make informed decisions regarding large volumes of collected records, automated processes are used to analyze and identify relationships that exist among the records. One of the most useful processes for gaining a quick and useful understanding of these relationships is the use of data analysis visualization tools for visualization-based exploratory data analysis. Data analysis visualization tools are software packages that create graphical representations of the relationships that exist among data sets or records.
Visualization-based exploratory data analysis involves three primary steps: importing the data from one or more sources, creating the visualizations from that data, and exploring and mining the data. A common problem for data analysis systems is getting data into a form that can be handled appropriately by the analysis software. The data streams and formats that are encountered are as varied as the creators of that information. Thus, it has been difficult to provide an input process or system to accommodate the variety of data formats.
These difficulties are amplified if analysis is desired for information of many types. For example, free text, numeric, categorical information, and genomic sequence data are different types of data that may be encountered in records. In addition to desiring systems that can handle data of various types, users desire flexibility in transforming data from one format to another. Depending on the nature of the data, the user may desire a variety of mathematical or other operations to be performed on the information, such as normalization schemes. This is particularly important with experimentally derived data.
There are several standard data formats, such as tab-delimited ASCII files, that can be read by spreadsheet and other programs. Automated and semi-automated methods exist for importing data from these standard formats into software analysis packages (e.g., the Text Import Wizard in Microsoft Excel). However, even within these standard formats, the organization of information may not be consistent with the desired, ensuing data analysis.
R. Ford, R. Thompson and D. Thompson, Supporting heterogeneous data import for data visualization, Proceedings of the 1995 ACM Symposium on Applied Computing, pp.81-85, 1995, have described one approach for combining datasets from various sources and formats. They use format converters that can be created for preexisting data formats and apply these to convert digital geographic information into a format that can be used by IBM Data Explorer software. This approach has general utility, but the output types are not sufficiently structured for enabling integrated data visualization and mining across disparate data types and structures.
H. Davulcu, G. Yang, M. Kifer, and I. Ramkrishnan, Computational Aspects of Resilient Data Extraction from Semistructured Sources, Proceedings of the 19th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems, pp. 136-144, 2000, describe a method for extracting information from semistructured data sources such as HTML pages. The variability in construction of those data sources and the need to extract common attributes from those sources presents difficulties. The approaches rely on extraction expressions as a means for identifying target data within a larger context. U.S. Pat. No. 5,913,214 to Madnick, et al., entitled Data extraction from word wide web pages, also describes a system for extracting disparate, heterogeneous data from multiple semi-structured data sources. In this system, a data translator is added to provide the data context needed for queries and retrieval. In a related patent, U.S. Pat. No. 5,953,716, entitled Querying heterogeneous data sources distributed over a network using context interchange, methods are provided to generally handle problems in data set creation, but these methods do not handle unstructured data sources, do not provide means for integrating multiple data types, and do not permit user-defined manipulations of the data.
Another approach for combining information from different sources is discussed in U.S. Pat. No. 6,023,694 to Kouchi, et al., entitled Data retrieval method and apparatus with multiple source capability. In the ""694 patent, information is retrieved from two or more data sources by using drivers specific for each data format and converted to a database with a standardized data structure using key categories for organizing the data. U.S. Pat. No. 5,721,912 to Stepczyk, et al., entitled Graphical user interface for creating database integration specifications, also uses a data translation step in the framework of a graphically represented workflow.
C. Squire, Data Extraction and Transformation for the Data Warehouse, Proceedings of the 1995 ACM SIGMOD International Conference on Management of Data, pp. 446-447, 1995, discusses the integration and transformation of data for a data warehouse. However, Squire does not provide a general process for importing data into visualization and mining systems and does not describe a system that supports integration of different data types.
To enable analytical and visualization methods to be applied to multiple data types the final data format is standardized. D. King, Using Dimensions to Represent Data Base Attributes, Conference Proceedings on APL and the Future, pp. 151-155, 1985; and G. Bringham and E. Shaw, An N-Dimensional Data Structure in Support of Electronic Data Interchange (EDI) Translation, Proceedings of the International Conference on APL ""91, pp. 71-79, 1991, describe a standard database data structure in which each data object can be thought of as a collection of attributes. As such, each object may be represented by an N-dimensional collection of elements. The discussed N-dimensional collection is a valuable construct for data visualization, but does not have the flexibility to incorporate data from multiple sources, the ability to manipulate the data, and the ability to represent some data types (e.g., genomic sequence or text).
U.S. Pat. No. 5,361,326 to Aparicio et al., entitled Enhanced interface for a neural network engine, describes one approach for bringing together some aspects of the data import requirements. A generalized data translator is provided along with an ability to manipulate or transform the raw data as part of the front end to a neural network analysis.
After importing data for analysis, a number of manipulations may be made. These include operations or transformations with the data or merging overlapping information. In order to process these steps correctly, the data type for those operations should be defined. The data type can be defined by the user at the time of the process or an automatic method can be used for identifying data type. One method for identifying data type is described in U.S. Pat. No. 5,991,714 to Shaner, entitled Method of identifying data type and locating in a file. The ""714 patent uses an n-gram-based approach to characterize an electronic file of unknown data type. The n-grams are weighted and compared to defined thresholds for known data types. Automatic data type determination is also described in U.S. Pat. No. 6,014,661 to Ahlberg, et al., entitled System and method for automatic analysis of data bases and for user-controlled dynamic querying. The data is subjected to a series of tests that assess consistency with defined data types. For example, if the data contains only alphabetic ASCII characters, the data is assumed to be text or if the values detected are limited to 0 and 1, the data is assumed to be binary.
Data type recognition has also been applied to data streams as discussed in U.S. Pat. No. 5,784,544 to Stevens, entitled Method and system for determining the data type of a stream of data. The ""544 patent approach sequentially assumes each of the valid data types, translates the data from one type to another, and uses a count of valid characters as the basis for data type determination. This method applies to character type directly as opposed to the category of data (numerical, text, categorical, etc.). However, these data type identification techniques do not facilitate importation of data sets of various types and formats for use in a form readily usable by a visualization based data analysis system.
Thus, a system or method is needed that enables the importation of data of various types into a data visualization and analysis system and that provides flexibility in producing a final data set for analysis from differently formatted data sets.
Generally described, methods and systems consistent with the present invention import data into a data analysis system by enabling access to data from multiple data sources, where the data of the data sources are often structured in different data formats, and by providing an interface for importing data into the data analysis system. The interface enables a user to customize the formatting of the data as the data is being processed for importation into the data analysis system.
More particularly, a method or system of importing data into a data analysis system, consistent with the present invention, enables a user to specify first user defined options for operating on a first data set received during a data importation process. An intermediate representation of the data set is generated during the data importation process based on the user first defined options. A user may specify second user defined options based on the intermediate representation during the data importation process. The second user defined options are processed to produce a final data representation of the data set to be used for analysis of the data. This may be extended to more than two data sets, combining them in any order or simultaneously.
The intermediate representation may be a data table. The second user defined options may enable a user to select particular rows and columns of the data set for inclusion in the final data representation. The processing of a data set may include merging a first and second data set to produce the final data representation. The second user defined options may enable a user to select a basic operation for combining the data sets or to select a non-basic operation for combining the data sets. The basic operation may combine data sets in response to a user""s selection of a first graphical interface control, and the non-basic operation may combine the data sets based on user selection of at least two graphical interface controls from a group of graphical interface controls.
Another aspect consistent with the present invention provides a method of processing data sets for use in a data analysis system. The format of a data set to be imported into the system is determined and each data location of the data set that has a value missing is assigned a missing value indicator. The indicator may be the same indicator for each missing value of each data set processed by the system or missing value indicators may be uniquely chosen for each data set based on user selection.